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GLOSSARY 


Characterization 


Classification 


Detection 


Discrimination 


False  Alarm  (Detection) 

False  Alarm  (Discrimination) 

Target  of  Interest 


Determination  of  parameters  that  are  intrinsic  to  a  target  and  can  be 
used  to  make  a  discrimination  decision 

Formally,  determination  that  an  object  belongs  to  a  particular  class 
of  ordnance  (i.e.,  is  a  155  as  opposed  to  an  81).  Classification,  by 
its  formal  definition,  will  not  be  explored  in  this  study.  Instead,  in 
this  document  we  will  use  the  term  classification  as  a  synonym  for 
discrimination. 

Determination  of  the  presence  of  a  target,  typically  by  observation 
of  a  signal  level  crossing  a  threshold  set  to  limit  the  probability  that 
a  crossing  would  be  caused  by  noise  or  interference. 

Determination  that  a  detected  object  is  (1)  high  confidence  clutter 
(and  need  not  be  dug)  or  (2)  ordnance  or  unknown  (and  must  be 
removed) 

Declaration  of  a  target  that  is  actually  caused  by  noise,  interference, 
geology 

Declaration  of  an  item  that  could  safely  be  left  in  the  ground  as 
ordnance  or  unknown 

For  this  study,  defined  to  be  intact  munitions,  both  HE  and  practice; 
sizeable  pieces  of  munitions  (on  the  order  of  half  a  round);  and 
items  that  look  like  munitions  (e.g.,  pipes  of  similar  size) 
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1.0 


EXECUTIVE  SUMMARY 


1.1  BACKGROUND 

With  current  technology  and  survey  practiees,  unexploded  ordnance  (UXO)  site  eharacterization 
is  ineffieient  and  ineomplete.  Not  all  buried  UXO  are  routinely  deteeted,  and  those  that  are 
cannot  be  routinely  distinguished  from  other  items  in  the  ground  that  pose  no  risk.  The  impaet  on 
the  Department  of  Defense  (DoD)  is  elear — and  expensive.  A  2003  report  by  the  Defense 
Seienee  Board  observed:  “The  . . .  problem  is  that  instruments  that  ean  deteet  the  buried  UXOs 
also  deteet  numerous  serap  metal  objeets  and  other  artifaets,  whieh  leads  to  an  enormous  amount 
of  expensive  digging.  Typieally  100  holes  may  be  dug  before  a  real  UXO  is  unearthed!” 

Over  the  past  deeade,  DoD  has  invested  heavily  in  developing  survey  data  analysis  and 
proeessing  teehniques  for  use  with  eommereial  sensors  that  ean  improve  UXO  deteetion  and 
diserimination  between  UXO  and  elutter.  These  techniques  include  eharaeterization  proeedures 
for  estimating  target  features  from  survey  data  (size,  shape,  depth  of  burial,  orientation,  etc.)  and 
feature-based  classifieation  proeedures  to  aid  deeision  making. 

Our  teehnieal  approaeh  promotes  the  selection  of  potential  UXO  targets  using  quantitative 
evaluation  eriteria  and  transparent  deeision-making  proeesses.  As  sueh,  we  developed  UX- 
Analyze,  an  analysis  framework  within  Oasis  montajTM  that  integrates  quantitative  analysis 
algorithms  and  custom-designed  visualization  schemes.  UX-Analyze  was  eoneeived,  eoded,  and 
validated  under  MM-0210.  The  analysis  algorithms  provide  quantitative  evaluation  eriteria. 
Speeifically,  they  assume  a  dipolar  souree  and  derive  the  best  set  of  induced  dipole  model 
parameters  that  aeeount  for  the  spatial  variation  of  the  signal  as  the  sensor  is  moved  over  the 
objeet.  The  model  parameters  are  target  loeation  and  depth,  three  dipole  response  eoeffieients 
corresponding  to  the  principal  axes  of  the  target  (eleetromagnetie  induction  [EMI]  only),  and  the 
three  angles  that  deseribe  the  orientation  of  the  target.  The  souree ’s  size  ean  be  estimated  using 
empirieal  relationships  between  either  the  dipole  moment  for  magnetie  data  or  the  sum  of  the 
targets’  response  eoeffieients. 

1.2  OBJECTIVES  OF  THE  DEMONSTRATION 

The  objeetive  of  this  teehnology  demonstration  was  to  discriminate  4.2  ineh  mortars  from  native 
elutter  at  Camp  Sibert,  Alabama,  by  eharaeterizing  and  elassifying  anomalies  identified  in  EMI 
and  magnetie  survey  data.  Miselassifying  a  target  of  interest  as  an  item  that  ean  be  left  in  the 
ground  (viz.,  a  false  negative)  was  defined  to  be  the  primary  failure.  The  objectives  were 
sueeessfully  aehieved  and  doeumented. 

1.3  REGULATORY  DRIVERS 

A  stated  objeetive  in  the  DoD’s  Report  to  Congress  in  2001  is  to,  among  other  things,  develop 
standards  and  protoeols  for  navigation,  geoloeation,  data  aequisition,  data  proeessing,  and 
performance  of  UXO  teehnologies,  including  software  and  visualization  tools  needed  to  provide 
regulatory  and  publie  visibility  to  and  understanding  of  the  analysis  and  deeision  proeess  made  in 
UXO  response  aetivities. 
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1,4  DEMONSTRATION  RESULTS 


We  analyzed  870  EM61  MK2  array  anomalies  and  969  magnetie  anomalies.  We  based  the 
diserimination  decision  on  the  net  polarizability  (Sb)  for  EMI  and  on  the  inverted  magnetic 
moment  for  magnetic  anomalies  (the  two  data  sets  were  interpreted  independently).  Eabeled  data 
from  a  geophysical  prove-out  and  from  truthed  anomalies  within  the  survey  area  was  used  to 
establish  decision  boundaries. 

At  our  chosen  operating  threshold  for  the  EMI  data,  we  correctly  classified  100%  of  the  4.2  inch 
mortars  (Pd  of  1.0)  and  correctly  classified  49%  of  the  remaining  non-targets  of  interest  (TOI)  as 
high  confidence  clutter  that  could  safely  remain  buried. 

At  our  chosen  operating  threshold  for  the  magnetic  data,  we  correctly  classified  100%  of  the 
4.2in  mortars  (probability  of  detection  [Pd]  of  1.0)  and  correctly  classified  43%  of  the  remaining 
non-TOI  targets  as  high  confidence  clutter  that  could  safely  remain  buried. 

1.5  STAKEHOLDER/END-USER  ISSUES 

Discrimination  decisions  can,  in  general,  be  legitimately  based  on  a  variety  factors,  including 
estimates  of  size,  burial  depth,  remnant  magnetization,  decay  rate,  and  the  relative  magnitudes  of 
the  inverted  magnetic  polarizations.  Of  these  factors,  the  estimated  size  is  perhaps  the  most 
robust  to  minor  data  errors,  such  as  positioning  or  orientation  glitches,  while  the  magnitudes  of 
the  recovered  magnetic  polarizations  are  the  least.  This  demonstration  showed  that  is  possible  to 
separate  4.2  inch  mortars  from  native  clutter  found  at  Camp  Sibert  using  size  estimates  derived 
from  geophysical  data  that  was  acquired  in  survey  mode. 
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2.0  TECHNOLOGY  DESCRIPTION 


2.1  TECHNOLOGY  DEVELOPMENT  AND  APPLICATION 

The  anomaly  characterization  algorithms  embedded  in  UX-Analyze  assume  a  dipolar  source  and 
derive  the  best  set  of  induced  dipole  model  parameters  that  account  for  the  spatial  variation  of 
the  signal  as  the  sensor  is  moved  over  the  object.  The  model  parameters  are  target 
location,  three  dipole  response  coefficients  corresponding  to  the  principal  axes  of  the  target  and 
the  three  angles  that  describe  the  orientation  of  the  dipole.  The  size  of  the  target  can  be  estimated 
from  the  sum  of  the  targets’  response  coefficients.  The  shape  of  the  target  can  be  estimated  from 
the  relative  magnitudes  of  the  three  coefficients.  For  example,  cylindrical  objects,  like  most 
unexploded  ordnance  (UXO),  have  one  large  coefficient  and  two  smaller,  equal  coefficients. 
Platelike  objects  nominally  have  two  large  and  one  small  coefficient.  Spherical  objects  have 
three  roughly  equal  coefficients. 

UX-Analyze  was  developed  to  allow  users  to  systematically  identify,  extract,  edit,  and  store  data 
around  individual  anomalies.  It  provides  efficient  data  structures  and  access  for  the  analysis 
algorithms,  stores  the  fitted  parameters,  and  allows  for  multiple  data  types  and  surveys.  This 
module  is  the  interface  between  Oasis  montaj  and  the  demonstration  analysis  software 
(Figure  1). 


Figure  I,  UX-Analyze  Screen  Snapshot  During  Data  Analysis, 


Characterization  routines  for  magnetic  and  EMI  data  have  been  integrated  with  UX-Analyze 
framework.  These  3-D  routines  include  graphic  displays  and  controls  that  allow  the  user  to 
manually  select  and  filter  the  input  data  for  each  anomaly  (Figure  2). 
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Figure  2,  Screen  Snapshots  Showing  the  User  Interface  During  Data  Inversion, 


The  measured  data  is  shown  in  the  left  map,  the  model  parameters  are  displayed  in  the  lower 
center  window,  and  the  forward  model  generated  using  the  model  parameters  is  shown  in  the 

right  map. 

UX-Analyze  produces  individualized  anomaly  reports,  one  for  each  anomaly,  to  document  the 
decision  process  for  each  anomaly  (Figure  3).  In  each  plot,  the  measured  data  is  graphically 
displayed  next  to  the  modeled  data.  The  model  parameters  are  listed  in  the  middle  of  each  page, 
and  a  profile  extracted  along  the  transect  that  is  the  closest  to  the  inverted  dipole  location  is 
located  at  the  bottom. 

Essentially,  the  anomaly  plots  graphically  provide  an  intuitive  confidence  measure.  If  the 
measured  and  modeled  data  are  indistinguishable,  the  reviewer  can  have  confidence  that  the 
estimated  source  parameters  are  approximately  correct.  If  the  two  maps  are  do  not  resemble  each 
other,  however,  it  tells  us  that  the  source  in  question  (1)  cannot  be  represented  well  using  a  point 
dipole  source,  (2)  is  not  isolated,  (3)  does  not  have  sufficient  signal-to-noise  ratio,  or  (4)  was  not 
properly  sampled  (spatially  or  temporally).  In  any  case,  if  the  two  maps  are  dissimilar  the 
inverted  model  parameters  are  most  likely  not  correct. 

This  demonstration  utilized  magnetic  and  EMI  array  data  acquired  at  Camp  Sibert  [12].  The  data 
collection  hardware  consisted  of  a  low-magnetic-signature  vehicle  that  is  used  to  tow 
magnetometer  and  EMI  arrays  over  large  areas  (10-25  acres/day)  to  detect  buried  UXO  (Figure 
4).  Spatial  registration  information  is  recorded  using  high  performance  real-time  kinematic 
(RTK)  Global  Positioning  System  receivers  with  position  accuracies  of  -5  cm. 
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Measured  Data 
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Figure  3,  UX-Analyze  Generates  a  One-Page  Summary  for  Each  Anomaly, 

In  the  anomaly  summaries  shown  above,  the  measured  data  is  shown  in  the  upper  left  hand 
comer,  the  inverted  model  parameters  in  the  middle  left,  the  forward  model  in  the  upper  rights, 
and  a  profile  in  the  lower  left  comer.  This  layout  was  selected  to  provide  insight  into  the 
confidence  of  the  analysis  and  conclusions.  EMI  data  for  the  anomaly  are  shown  in  the  left 

summary,  and  magnetic  data  on  the  right. 


Figure  4,  Naval  Research  Laboratories  Multisensor  Towed  Array  Detection  System 

(left:  magnetomer  array,  right:  EM61  MK2  array). 

2.2  PROCESS  DESCRIPTION 

We  received  the  final  geopositioned,  demedianed  magnetometer  and  EM61  MK2data  from  the 
ESTCP  Program  Office.  We  inverted  each  anomaly,  taking  care  to  limit  competing  signatures 
and  extraneous  noise,  using  physics-based  dipole  models.  Pitted  model  parameters  include 
anomaly  size  (based  on  the  moment  for  magnetic  data  and  the  trace  of  the  polarizability  tensor 
for  EMI),  shape  (EMI  only),  XY  position,  depth,  orientation,  and  fit  error  statistics.  In  addition  to 
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presenting  the  results  of  the  inversion  in  spreadsheet  form,  UX-Analyze  generated  an  anomaly 
summary  sheet  that  shows  the  measured  data,  inversion  results,  and  model  data  for  quality 
eontrol  (QC)  purposes. 


TOIs  were  defined  to  include  (1)  intact  munitions,  both  high  explosive  (HE)  and  practice, 
(2)  sizeable  pieces  of  the  munitions  of  interest,  on  the  order  of  half  a  round,  and  (3)  items  that 
look  like  munitions  (i.e.,  pipes  of  similar  size). 

Items  that  may  be  safely  left  in  the  ground  included  HE  fragments,  single  fins,  cultural  debris, 
geology,  and  small  arms  rounds  (if  present). 

The  demonstration  focus  was  to  identify  items  that  may  be  safely  left  in  the  ground.  The  main 
failure,  therefore,  was  misclassijying  a  TOI  as  an  item  that  can  be  left  in  the  ground.  Individual 
targets  were  classified  as  indicated  in  Table  1.  These  discrimination  labels  were  then  used  to 
calculate  the  rolled  up  performance  measures  presented  in  Section  4. 

Table  1.  Discrimination  Labels  (Categories), 


Category 

Comment 

Recommended  Action:  (Dig 
or  Leave  in  Place) 

1 

High  confidence  clutter 

Leave  in  Place 

2 

Low  confidence  clutter 

Dig 

3 

Can’t  analyze 

Dig 

4 

Low  confidence  TOI 

Dig 

5 

High  confidence  TOI 

Dig 

The  “Can’t  Analyze”  Category  3  label  was  used  to  identify  targets  that  could  not  be  inverted 
(i.e.,  the  inversion  did  not  converge)  or  could  be  inverted  but  the  fit  error  statistics  indicated  that 
the  results  could  not  be  trusted.  We  determined  the  threshold  for  the  later  case  for  each  dataset 
individually  using  labeled  data. 

2,3  PREVIOUS  TESTING  OF  THE  ANALYSIS  TECHNOLOGY 

Our  analysis  framework  and  procedures,  described  in  a  number  of  publications  [2-9],  have  been, 
or  are  being,  successfully  demonstrated  in  a  variety  of  ESTCP  Projects,  including  MM-9811, 
MM-9812,  MM-199526,  MM-9812,  MM-9918,  MM-0031,  MM-0033,  MM-0034,  and  MM- 
0108. 

Previous  testing  of  the  analysis  approach  and  software  suite  included  a  shakedown  using  a  large 
volume  dataset  acquired  at  Standardized  UXO  Technology  Demonstration  Site,  Aberdeen 
Proving  Ground  Maryland  in  2006.  During  the  shakedown  test,  we  encountered  programming 
bugs  and  logic  problems  that  had  not  been  exposed  during  development  or  tests  with  more 
modest  data  sets.  We  identified  and  resolved  71  bugs  associated  with  data  handling,  51  bugs 
associated  with  visualization,  15  bugs  associated  with  the  inversion  routines,  22  bugs  that  were 
generated  while  adding  new  features  and  user  interfaces,  and  38  bugs  associated  with 
miscellaneous  functionality.  At  the  conclusion  of  the  shakedown  test,  all  identified  problems  had 
been  fixed. 
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Technology  demonstrations  were  also  carried  out  at  live  sites  in  Connecticut  and  Colorado. 
Demonstration  plans  and  interim  reports  are  on  file  with  the  ESTCP  Program  Office. 

2,4  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

This  demonstration  technology  uses  spatially  referenced  geophysical  data  to  estimate  target 
features  using  a  dipole  model.  This  has  an  inherent  advantage  over  ancillary  analysis,  less  robust 
methods  commonly  employed.  Ancillary  analysis  methods  sometimes  include  metrics  such  as 
the  anomaly  amplitude,  half  width,  spatial  footprint,  or  overall  “look.”  These  later  metrics  are, 
however,  sensitive  to  the  targets’  orientation  and  depth  of  burial.  The  methodology  demonstrated 
here  separates  the  measured  signatures  into  that  which  is  inherent  to  the  target,  and  that  which  is 
related  to  the  geometry  of  the  problem  (such  as  distance  to  sensor  and  orientation). 

The  primary  advantage,  therefore,  is  the  potential  for  discriminating  between  UXO  and  non- 
UXO-like  objects  based  on  quantified  metrics  calculated  by  inverting  geophysical  survey  data. 
Results  from  past  demonstration  have,  in  fact,  shown  that  that  discrimination  is  possible  using 
magnetic  and  electromagnetic  data  [10].  Magnetic  discrimination  is  based  primarily  on  the 
apparent  fitted  dipole  size  (or  scaled  dipole  moment).  Using  EMI  data,  increased  discrimination 
performance  can  sometimes  be  achieved  by  utilizing  estimated  shape  information.  If  successful 
discrimination  capabilities  can  be  achieved,  significant  excavation  savings  can  be  realized  by 
leaving  the  nonhazard  clutter  items  unearthed  or  by  altering  the  way  in  which  they  are  recovered. 

Known  limitations  to  the  data  analysis  approach  adopted  here  result  from  non-unique  inversion 
results,  and  overlapping,  or  non  distinct,  signatures  in  feature  space.  The  former  limitation,  one 
in  which  multiple  sets  of  model  parameters  explain  the  vast  majority  of  the  observed  data,  is  well 
known.  The  second,  while  perhaps  not  as  widely  appreciated,  is  equally  problematic.  Inverting 
EMI  data  using  dipole  models  results  in  three  eigenvalues  of  the  magnetic  polarizability  tensor, 
each  corresponds  to  a  principal  axis  of  object.  Discrimination  is  possible  only  to  the  degree  that 
the  derived  eigenvalues  are  different  for  different  objects  and  stable  for  similar  classes  of  objects. 
In  other  words,  even  with  ideal  data,  the  estimated  burial  depth,  apparent  size,  and  shape  features 
may  not  separate  UXO  and  clutter  signatures  into  distinct,  non-overlapping  classes.  This  is 
because  the  anomaly  features  derived  from  EMI  and  magnetic  data  are  not  unique  to  UXO. 
Clutter  items  that  have  similar  shapes  and  burial  attributes  to  ordnance  can  have  geophysical 
signatures  that  are  indistinguishable  from  UXO  signatures  and,  as  such,  will  have  similar 
eigenvalues  and  therefore  likely  be  classified  as  ordnance.  Examples  include  items  such  as  pipes, 
post  sections,  and  axial  symmetrical  fragments. 
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3.0  DEMONSTRATION  DESIGN 

3.1  PERFORMANCE  OBJECTIVES 

Table  2,  Performance  Objectives  for  tbe  Discrimination  Study. 


Type  of 
Performance 
Objective 

Primary  Performance 
Criterion 

Expected  Performance 
(Metric) 

Actual 
Performance 
Objective  Met? 

Quantitative 

Correct  characterization 
parameters 

X,Y  distance:  <20  cm  (Is)  Z: 

<15  cm  app  size:  (COV*  <0.3) 

Yes 

Maximize  correct  determinations 
of  nonordnance 

Discrimination  eliminates  >20% 
of  the  detections  corresponding 
to  items  not  of  interest  at  the  no¬ 
dig  threshold 

Yes 

Maximize  correct  determinations 
of  ordnance 

Discrimination  retains  all 
detected  objects  of  interest  above 
the  no-dig  threshold  (may  be 
multiple  thresholds) 

Yes 

Demonstrate  effect  of  data 
quality  (line  spacing,  location 
error,  signal/noise  levels,  etc.)  on 
the  ability  to  correctly  extract 
parameters  and  classify  ordnance 
versus  nonordnance 

Data  quality  effects  can  be 
isolated  and  understood 

Yes 

*coefficient  of  variation  (COV)  -  standard  deviation/root  mean  square  (RMS)  error 


3.2  SELECTING  THE  TEST  SITE 

Camp  Sibert  is  located  within  the  boundaries  of  Site  18  of  the  former  Camp  Sibert.  The  land  is 
under  private  ownership  and  is  used  as  a  hunting  camp.  The  eriterion  that  drove  the  site  selection 
process  were  (1)  a  single  use  artillery  or  mortar  range,  (2)  simple  elutter  environment,  (3)  benign 
geology,  (4)  live  ordnanee  used,  and  (5)  benign  topography  and  vegetation.  Additional 
considerations  were  size  (20-25  aeres  was  desired),  anomaly  density  (mostly  isolated  anomalies, 
100-200  per  aere),  total  anomaly  eount  (2,500  to  5,000  anomalies  were  desired),  and 
aeeess/authorization  to  seed  site  with  inerted  targets. 

3.3  TEST  SITE  HISTORY/CHARACTERISTICS 

Information  on  the  former  Camp  Sibert  is  available  in  the  archival  literature  sueh  as  an  Archives 
Seareh  Report  (ASR)  developed  in  1993.  Camp  Sibert  is  eomposed  of  mainly  sparsely  inhabited 
farmland  and  woodland  and  eneompasses  approximately  37,035  aeres.  The  Gadsden  Munieipal 
Airport  occupies  the  former  Army  airfield  in  the  northern  portion  of  the  site.  The  site  is  loeated 
approximately  50  miles  northwest  of  the  Birmingham  Regional  Airport  or  86  miles  southeast  of 
the  Huntsville  International  Airport.  The  site  is  near  exit  181  off  Interstate  59  in  Gadsden  and 
loeated  approximately  8  miles  southwest  of  the  City  of  Gadsden,  near  the  Gadsden  Munieipal 
Airport. 
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3.4  PHYSICAL  SETUP  AND  OPERATION 


This  data  analysis  demonstration  utilized  data  aequired  during  the  ESTCPs  UXO  Diserimination 
Pilot  Program,  whieh  was  eondueted  in  2007.  As  sueh,  others  were  responsible  for  the  signifieant 
tasks  of  establishing  a  geophysical  prove  out,  seeding  control  targets,  acquiring  the  geophysical 
data,  overseeing  recovery  operations,  and  managing  the  information  flow.  These  activities  are 
being  reported  directly  by  the  ESTCP  Program  Office.  Key  chronological  dates  with  regard  to 
the  overall  program  are  listed  in  Table  3. 

Table  3,  Timetable  of  Demonstration  Events. 


Event 

Date 

Data  collection  surveys 

ApriEMay  2007 

Receipt  of  data  from  Program  Office 

June  18,  2007 

Receive  target  lists  of  anomalies 

June  29,  2007 

Receive  labeled  data  for  training 

August  6,  2007 

Submit  letter  report  regarding  analysis  approach  and  expected  performance 

August  22,  2007 

Submit  letter  report  regarding  analysis  approach  and  expected  performance — revision 

September  14,  2007 

Submit  analysis  results  (prioritized  dig  list) 

September  25,  2007 

Received  performance  results  from  IDA* 

November  20,  2007 

In-Progress  Review 

November  30,  2007 

ESTCP/SERDP  Symposium  Brief 

December  4,  2007 

*Institute  for  Defense  Analysis 
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4.0  PERFORMANCE  ASSESSMENT 


4.1  PERFORMANCE  DATA 

Scoring  performances  for  our  analysis  of  the  EMI  sensor  data  are  reported  in  Table  4  and  shown 
graphically  in  Figure  5.  A  receiver  operating  characteristic  (ROC)  chart  is  shown  in  Figure  6. 
From  a  declared  eategory  perspeetive,  the  Category  3  targets  are  plotted  first,  followed  by 
Categories  5  (red),  4  (orange),  2  (yellow),  and  finally  1  (green). 

Using  the  established  thresholds,  there  were  no  classification  failures.  All  UXO  were  categorized 
medium  or  high  confidenee  UXO  (Category  4  or  Category  5),  while  eorrectly  classifying  49%  of 
the  total  digs  as  objeets  that  eould  be  safely  left  in  the  ground  (Category  1). 

Table  4,  Performance  Summary:  EM61  MK2, 


EM61  Array  UXA 


Cultural 

Munition 

Debris 

No  Contact 

Rock 

Soil 

UXO 

1 

75 

155 

2 

22 

101 

0 

2 

8 

59 

1 

3 

11 

0 

3 

17 

16 

0 

18 

44 

0 

4 

4 

28 

1 

1 

7 

6 

5 

2 

34 

1 

1 

4 

113 

TOTAL 

106 

292 

5 

45 

167 

119 

EIII61  MK2  Array  Scoang  Perfonnai>oe  (UXA) 
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Figure  5,  EM61  MK2  Array  Performance  as  a  Function  of  Classification  Category, 
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cSAICEMBI  MK2  ARRAY  SUBMITTAL  UXASC070925CSV>  43  934  534  497  efn61,  Test  Al 


Figure  6,  EM61  MK2  Array  ROC  Chart 

(Analysis  based  on  size  metric). 

Scoring  performances  for  the  magnetometer  (mag)  analysis  are  reported  in  Table  5  and  shown 
graphically  in  Figure  7.  A  ROC  chart  is  shown  in  Figure  8.  Using  the  thresholds  adopted  for  this 
analysis,  there  were  no  false  positives,  while  eliminating  43%  of  the  total  digs. 

Table  5,  Performance  Summary  Mag  Array 


Mag  Array 


Cultural 

Munition 

Debris 

No  Contact 

Rock 

Soil 

uxo 

1 

60 

258 

2 

18 

22 

0 

2 

4 

41 

1 

3 

12 

3 

3 

35 

63 

2 

42 

56 

0 

4 

1 

33 

0 

3 

15 

45 

5 

1 

23 

1 

2 

8 

70 

TOTAL 

101 

418 

6 

68 

113 

118 
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MAG  MK2  Array  Scoring  Performance 
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Figure  7,  Mag  Array  Performance  as  a  Function  of  Classification  Category, 


«5AICMAGSUBMITTAl.UXASCO70925CSV>,  13«  13*0  1341  1337,  m»g,  Test  A1 


4.2  PERFORMANCE  CRITERIA 

Our  prioritized  dig  lists  were  scored  against  the  emplaced  targets  by  analysts  from  the  Institute 
for  Defense  Analyses.  Performance  results  for  each  of  our  six  prioritized  dig  lists  are  shown  in 
Tables  6  and  7. 
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In  Tables  6  and  7,  the  ROC  caleulation  exeludes  Category  3  items  beeause  they  were,  by  default, 
reeommended  to  be  removed  and  therefore  do  not  contribute  any  gains  due  to  the  discrimination 
process.  Also,  the  metric  used  for  the  estimation  of  size  is  the  coefficient  of  variation  (COV), 
which  is  simply  the  standard  deviation  divided  by  the  root  mean  square  (RMS)  value  of  the  size 
parameter.  It  essentially  reports  variability  in  the  estimated  size  parameter. 

Table  6,  Performance  and  the  Confirmation  Methods:  EM61  MK2  Array 


Performance 

Metric 

Expected  Performance 
(pre  demo) 

Performance 
Confirmation  Method 

Actual  Performance 
(post  demo) 

Pdisc  (emplaced)  at 
Pthreshold 

>95% 

Comparison  to  seeded  items 

1.0 

Pfa  at  100%  Pdisc 

<80%  of  FA  remain 

Validation  digging 

0.270 

ROC 

0.4 

Gains  over  chance  diagonal 
Validation  digging 

0.470 

Accuracy  of 

parameter 

estimation 

X,Y  Distance:  <20  cm  Z:  <15 
cm  size  (UXO  only):  COV<0.3 

Comparison  to  truth  data 
Comparison  to  truth  data 
Comparison  to  truth  data 

18cm  (12cm/s)  20cm 
(17cm/s)  0.28 

Pdisc  =  probability  of  correct  discrimination 
Pfa  =  probability  of  false  alarm 


Table  7.  Performance  and  the  Confirmation  Methods:  Magnetic  Array 


Performance 

Metric 

Expected  Performance  (pre 
demo) 

Performance 
Confirmation  Method 

Actual  Performance 
(post  demo) 

Pdisc  (emplaced)  at 
Pthreshold 

>95% 

Comparison  to  seeded  items 

1.0 

Pfa  at  100%  Pdisc 

<80%  of  FA  remain 

Validation  digging 

0.486 

ROC 

0.4 

Gains  over  chance  diagonal 
Validation  digging 

0.416 

Accuracy  of 
parameter  estimation 

X,Y  Distance:  <20  cm  Z:  <15  cm 
size  (UXO  only):  COV<0.3 

Comparison  to  truth  data 
Comparison  to  truth  data 
Comparison  to  truth  data 

14cm  (lOcm/s)  10cm 
(12cm/s)  0.24 

4,3  DATA  ASSESSMENT 

A  performance  comparison  of  results  for  magnetic  and  EMI  data  are  shown  in  Figure  9. 
Although  both  approaches  performed  well  (Table  6  and  Table  7),  the  EMI -based  results  had 
fewer  Can’t  Analyze  declarations  (Category  3)  and  better  separation  between  classes  than  the 
magnetic  (as  visually  observed  by  the  slope  of  the  curve  as  well  as  Area  Under  Curve  (AUC)  of 
0.97  for  EMI  versus  0.91  for  mag). 

The  sources  of  the  “Can’t  Analyze”  Category  3  declarations  turned  out  to  be  small  fragments, 
scrap  metal,  no  contact,  or  soil  (see  Figure  10  for  photographs  of  the  recovered  targets).  These 
anomalies  were  characterized  by  low  amplitude  responses  and  small  spatial  footprints  and  were 
not,  therefore,  well  characterized  by  our  models  (Figures  11  and  12).  The  largest  clutter  items 
were  the  half  shells,  which  were  presumably  created  when  the  4.2  inch  shell  casings  split  open  at 
impact. 
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Figure  9,  Performance  Comparison  of  Size-Based  Classification  for  Magnetic  and  EMI 

Array  Data. 


Half  Shells 


4.2”  Mortars 


Munitions  Detxis 


Figure  10,  Photographs  of  Objects  Recovered  from  the  Sihert  Demonstration  Area, 
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Figure  11,  Example  EMI  Anomalies  Classified  “Can’t  Analyze”  (Category  3), 


Fragments 
Scrap  Metal 
Soil 
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Figure  12,  Example  Magnetic  Anomalies  Classified  “Can’t  Analyze”  (Category  3), 
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4.4  TECHNOLOGY  COMPARISON 


After  the  excavations  were  complete,  803  clutter  items  were  collected  from  Camp  Sibert, 
renumbered  (to  protect  their  identity),  and  shipped  to  test  facilities  in  Blossom  Point, 
Massachusetts.  The  idea  was  to  collect  data  in  a  controlled  setting  in  order  to  collect  extremely 
high-quality  data  that  could  be  used  to  establish  discrimination  bounds  for  this  collection  of 
UXO  and  clutter.  By  recording  signatures  for  each  object  individually  at  optimal  coil-target 
separations,  the  in-air  controlled  tests  inherently  removed  all  positioning  errors,  eliminated  any 
possible  overlapping  signatures,  eradicated  motion  noise,  and  did  away  with  unwanted  soil 
response. 

Time-domain  EMI  data  were  collected  using  a  new  fixed-geometry  array  developed  under 
funding  from  MM-0601.  The  sensor  consists  of  25  transmit  (Tx)/receive  (Rx)  pairs  arranged  in  a 
5x5  fixed-separation  grid.  The  net  polarizability,  defined  here  as  Shi  for  t  =  0.04-0.06  ms  shows 
that  the  4.2  inch  mortar  target  strength  is  clearly  larger  than  any  clutter  item  (Figure  13).  This 
suggests  that  perfect  discrimination  is  possible  for  this  distribution  of  UXO  and  clutter  if  the 
deleterious  effects  of  poor  positioning,  survey  noise,  and  competing  geological  sources  are 
eliminated. 


Net  Polarizability  (0.04-0.06  msec) 

Figure  13,  In-Air  Target  Strength  (net  polarizability,  Spi  for  t  =  0,04-0,06  ms)  Suggests 
Perfect  Discrimination  is  Possible  for  this  Collection  of  TOIs  and  Clutter, 

(The  4.2  inch  mortar  target  strength  is  clearly  larger  than  any  clutter  item.) 
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5.0  COST  ASSESSMENT 


5.1  COST  REPORTING 

Cost  categories  for  the  overall  ESTCP  Discrimination  Pilot  Program  include  mobilization,  field 
survey,  data  analysis,  demobilization,  and  reporting. 

Cost  eategories  for  the  analysis  reported  herein  are  shown  in  Table  8. 

Table  8.  Demonstration  Costs, 


Cost  Category 

Details 

Unit  Cost  ($) 

Preprocessing 

Loading,  re-leveling,  filtering,  and  data 
handling  prior  to  inversion 

Mag  Array 

3,000 

EM61  Array 

4,000 

Anomaly  extraction 

Extracting  anomaly  signatures  from  site 
database 

Mag  Array 

4,000 

EM61  Array 

6,000 

Data  characterization 

Inverting  anomaly  data  for  model  parameters 

Mag  Array 

10,000 

EM61  Array 

15,000 

Data  classification 

Performing  feature-based  classification 

Mag  Array 

2,000 

EM61  Array 

3,000 

Documentation 

Preparing  dig  sheet 

Mag  Array 

2,000 

EM61  Array 

2,000 

Reporting 

Preparing  and  revising  technical  documents 

Demonstration  Plan 

7,500 

Demonstration 

Report 

15,000 

TOTAL 

73,500 

Roughly  900  anomalies  were  identified  in  the  magnetic  and  EMI  data.  The  estimated  eosts 
required  to  proeess  and  analyze  eomparable  data  (viz.,  a  data  set  that  is  leveled  with  isolated 
anomalies),  exeluding  one-time,  demonstration-related  eosts  such  as  experimentation, 
optimization,  non-routine  analysis  and  testing,  and  reporting  are  presented  in  Table  9. 

Table  9,  Cost  Tracking  Categories  for  SAIC  Data  Analysis. 


Cost  Category 

Details 

Sub  Category 

Hours 

Cost*  ($) 

Preprocessing 

Loading,  re-leveling,  filtering, 
and  data  handling  prior  to 
inversion 

Mag  Array 

60 

7,200 

EM61  Array 

60 

7,200 

Anomaly  extraction 

Extracting  anomaly  signatures 
from  site  database 

Mag  Array 

40 

4,800 

EM61  Array 

40 

4,800 

Data  characterization 

Inverting  anomaly  data  for 
model  parameters 

Mag  Array 

20 

2,400 

EM61  Array 

20 

2,400 

Data  classification 

Performing  feature-based 
classification 

Mag  Array 

16 

1,920 

EM61  Array 

16 

1,920 

Documentation 

Preparing  dig  sheet 

Mag  Array 

8 

960 

EM61  Array 

8 

960 

TOTAL 

34,560 

*@  $120  per  hour 
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6.0  IMPLEMENTATION  ISSUES 


6.1  COST  OBSERVATIONS 

The  analysis  approach  demonstrated  here  utilizes  the  spatial  distribution  of  the  measured 
magnetic  or  EMI  signatures.  As  such,  it  requires  high  signal-to-noise  data  with  a  high  degree  of 
spatial  precision  across  the  footprint  of  the  anomaly. 

The  costs  to  acquire  data  that  will  support  discrimination  decisions  are  higher  than  that  required 
if  the  goal  is  only  to  detect  the  presence  of  an  object.  The  factors  affecting  acquisition  costs 
relate  to  particulars  of  the  sensing  system,  spatial  registration  system,  the  target  objectives,  and 
the  site  environment.  Although  these  costs  are  not  the  focus  of  this  demonstration,  they  are  very 
important  to  the  ultimate  transferability  of  this  approach. 

The  analysis  costs  are  also  higher  if  attempts  are  made  to  quantitatively  discriminate  rather  than 
only  to  detect.  The  factors  affecting  analysis  time  are  significantly  affected  by  (1)  the  degree  to 
which  the  anomalies  are  spatially  separated,  (2)  the  number  of  anomalies,  and  (3)  the  amount  of 
geologic-related  signatures  that  have  wavelengths  similar  to  the  targeted  signatures.  The  data 
density  is  also  a  factor,  but  only  marginally  so  compared  to  the  factors  listed  above  because  it 
affects  computer  run  time  and  not  analysts’  labor. 

6.2  PERFORMANCE  OBSERVATIONS 

Discrimination  performance  is  measured  by  our  ability  to  characterize  and  classify  one  object 
from  another.  The  factors  that  affect  performance,  therefore,  relate  to  (1)  the  similarity  (in 
feature  space)  between  the  TOI  and  non-TOI,  (2)  our  ability  to  accurately  measure  the  responses, 
the  presence  of  signatures  that  spatially  interfere  or  otherwise  compete  with  the  UXOs  response, 
as  well  as  (3)  our  ability  to  quantitatively  characterize  and  classify  the  source  objects.  Many  of 
these  factors  are  not  under  our  direct  control. 

6.3  SCALE-UP 

There  are  no  critical  issues  with  regard  to  scaling  up  the  demonstration  costs  reported  here  to 
larger,  full-scale  implementations.  The  cost  categories  may  not,  however,  scale  linearly.  The 
factors  listed  in  Section  6.1  will  determine  which,  if  any,  cost  categories  dominate  future 
technology  deployments. 

6.4  OTHER  SIGNIFICANT  OBSERVATIONS 

There  are  many  technical  factors  that  can  affect  implementation  of  the  analysis  technology 
discussed  in  this  report.  As  mentioned  earlier,  the  analysis  approach  demonstrated  here  utilizes 
the  spatial  distribution  of  the  measured  magnetic  or  EMI  signatures.  As  such,  it  relies  on  accurate 
3-D  spatial  measurements  as  well  as  on  stable  geophysical  measurements.  The  attitude  of  the 
geophysical  sensor  is  also  critically  important  to  inverting  for  meaningful  model  parameters.  If 
the  data  going  into  the  inversion  routines  are  noisy  or  contain  systemic  problems,  the  final 
discrimination  decisions  will  not  be  acceptable. 
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6.5  LESSONS  LEARNED 


Discrimination  decision  between  TOI  and  targets  that  are  not  of  interest  ean  be  based  on  a 
variety  of  metrics.  Some  of  these  metries  inelude  estimates  of  shape,  size,  remnant 
magnetization,  decay  rate,  magnetie  erossover,  or  even  depth  of  burial.  One  of  the  most  robust 
features  is  the  estimated  size,  whieh,  at  this  site,  was  used  to  eorreetly  eliminate  49%  of  the 
seleeted  EMI  targets  without  ineurring  a  single  false  negative. 

6.6  END-USER  ISSUES 

Eneouraging  explieit  diserimination  deeisions  requires  a  reeognition  that  implieit  diserimination 
always  oecurs,  an  appreciation  for  and  acceptance  of  risk-based  decisions,  capable  eontractors, 
validated  teehnologies,  and  knowledgeable  buyers.  Beeause  there  are  many  players,  opinions, 
situations,  and  objeetives  within  the  UXO  eleanup  spaee,  multiple  demonstrations  and  eontinued 
proeess  and  teehnology  advaneements  are  needed. 

The  ESTCP  Program  Offiee  established  an  Advisory  Group  to  faeilitate  interaetions  with  the 
regulatory  eommunity  and  potential  end  users  of  UXO  diserimination  teehnology  as  part  of  the 
UXO  Diserimination  Pilot  Program.  Members  of  the  Advisory  Group  included  representatives  of 
the  U.S.  Environmental  Proteetion  Agency  (EPA),  State  regulators.  Corps  of  Engineers  officials, 
and  representatives  from  the  serviees.  Together,  they  defined  goals  for  the  UXO  Diserimination 
Pilot  Program  and  developed  Projeet  Quality  Objeetives  used  in  this  report. 

The  diserimination  mindset  that  should  be  promoted  is  one  that  eneourages  geophysical  service 
providers  to  deliver  a  dig  list  that  is  prioritized  aeeording  to  objeetives  defined  by  the  site’s 
stakeholders.  The  deeision  metries  and  thresholds  should  be  quantitative,  transparent,  and 
doeumented.  Stakeholder  priorities  eould  reasonably  relate  to  size,  depth  of  burial,  shape, 
material  type,  or  munitions  type(s).  Onee  the  prioritized  dig  list  is  delivered,  the  stakeholders 
deeide  the  ensuing  aetions.  Stakeholders  and/or  site  managers  ean  realize  finaneial  savings  by 
either  modifying  exeavation  proeedures  based  on  the  probability  of  being  a  TOI  or  by  deelaring 
that  no  further  action  is  required  for  speeifie  anomalies.  The  point  is  that  the  deeision  to  take 
aetion  always  remains  with  the  stakeholders — as  it  must,  if  the  diserimination  mindset  is  to  be 
aeeepted. 

6.7  APPROACH  TO  REGULATORY  COMPLIANCE  AND  ACCEPTANCE 

The  UXO  industry  is  a  regulatory  driven  industry.  At  this  time,  we  do  not  know  of  any  polieies 
that  require  the  use  of  diserimination  teehnologies  at  live  sites.  Reeent  reports  by  the  Defense 
Seienee  Board,  however,  elearly  reeognize  the  potential  for  eost  savings  if  reliable  and 
defensible  diserimination  deeisions  ean  be  realized.  Additionally,  the  Corps  of  Engineers, 
Huntsville,  have  modified  data  item  deseription  MR-005 -05 A  to  inelude  fields,  albeit  optional, 
that  ean  be  used  to  doeument  anomaly  speeified  estimates,  sueh  as  those  generated  by  an  inverse 
modeling  approaeh  as  reported  here. 
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